Add distributed backend (XCCL) #1105
Conversation
|
|
||
| include(${CMAKE_ROOT}/Modules/FindPackageHandleStandardArgs.cmake) | ||
|
|
||
| set(XCCL_ROOT $ENV{CCL_ROOT}) |
There was a problem hiding this comment.
How do you get CCL_ROOT? I think you cannot assume it will be set after oneccl source.
There was a problem hiding this comment.
It will auto set after source oneccl env, and I remember oneccl update not affect this flag.
src/xccl/ProcessGroupXCCL.cpp
Outdated
| "Not able to create/get " | ||
| "XCCL Communicator since the devices are empty "); | ||
| { | ||
| // todo: why do we need mutex here? |
There was a problem hiding this comment.
I think we followed the same code logic as NCCL, right?
| m.impl("recv_any_source_", recv_any_source_XPU); | ||
| m.impl("reduce_", reduce_XPU); | ||
| m.impl("broadcast_", broadcast_XPU); | ||
| m.impl("allreduce_", allreduce_XPU); |
There was a problem hiding this comment.
In this PR, we only have allreduce implemented. Then should we only register allreduce here?
Chao1Han
changed the title
| bool is_reduction_op = false) { | ||
| TORCH_CHECK( | ||
| !isFloat8Type(type) && is_reduction_op, | ||
| "Float8 dtypes are not currenlty supported for XCCL reductions"); |
There was a problem hiding this comment.
For non-reduction collective, please add mapping from FP8 to ccl data type.
src/xccl/ProcessGroupXCCL.cpp
Outdated
| {at::kDouble, ccl::datatype::float64}, | ||
| {at::kBFloat16, ccl::datatype::bfloat16}, | ||
| {at::kBool, ccl::datatype::uint8}, | ||
| // use for allgather |
|
LGTM |
|
@EikanWang Could you please help review this PR? |
cmake/Modules/FindXCCL.cmake
Outdated
|
|
||
| include(${CMAKE_ROOT}/Modules/FindPackageHandleStandardArgs.cmake) | ||
|
|
||
| set(XCCL_ROOT $ENV{ONEAPI_ROOT}/ccl/latest) |
There was a problem hiding this comment.
Does it mean the ONEAPI_ROOT is a must-to-have environment variable?
There was a problem hiding this comment.
No, we only require oneCCL source in building, so CCL_ROOT is a must-to-have environment variable. Updated in latest.
This pr just implement allreduce, so add simple test case. In the long term, once all operations are implemented, we will have one or two test files to validate the basic operations. Other tests, such as FSDP and DTensor, will directly reuse PyTorch's unit tests. |
Then, I suggest reusing PyTorch test cases by disabling some test cases that are not applicable. Pls. check with Daisy. |
Motivation:
As design illustrated in Intel distributed support RFC pytorch/pytorch#141741, Intel GPU distributed Backend integration in PyTorch torch-xpu-ops.
Design:
USE_XCCL is set to ON by default. Users can manually set it to OFF to disable XCCL compilation. The OneCCL path is first searched in /opt/intel/oneapi/ccl/latest. If not found, it uses the CCL_ROOT flag set by the user after sourcing OneCCL. The USE_C10D_XCCL variable is intended to align with other distributed backend environment variables.
Oneccl lib link to torch_xpu align with other distribute backend.